Training Machine Learning models involves multiple steps, it gets more complex and time consuming when the size of the data set for training is in the range of 100s of GBs. Data Scientists run through large number of experiments and research which includes testing and training large number of models. Kubeflow provides various ML capabilities to accelerate the training process and run simple, portable scalable Machine Learning workloads on Kubernetes. Model parallelism is a distributed training method in which the deep learning model is partitioned across multiple devices, within or across instances. When Data Scientists adopt Model parallelism there's also a need to share the large dataset across Machine Learning models.

Groundbreaking breakthroughs in medical research have been enabled through machine learning. This paper discusses how the latest generation of Intel processors can be used in Amazon Web Services public and private environments to make new discoveries using advanced machine learning resources. In this paper we demonstrate how to use publicly available bone marrow datasets to efficiently train machine learning models using the latest technologies from Amazon and Intel. This paper discusses a real-world use case where we use advanced artificial intelligence technologies to train a bone marrow cell classification machine learning model from publicly available data. This trained model is then optimized using Intel OpenVino for inference.

Cresta Intelligence, a California-based AI startup, makes businesses radically more productive by using Expertise AI to help sales and service teams unlock their full potential. Cresta is bringing together world-renowned AI thought-leaders, engineers, and investors to create a real-time coaching and management solution that transforms sales and increases service productivity, weeks after application deployment. Cresta enables customers such as Intuit, Cox Communications, and Porsche to realize a 20% improvement in sales conversion rate, 25% greater average order value, and millions of dollars in additional annual revenue. This post discusses Cresta's journey as they moved from a multi-cloud environment to consolidating their machine learning (ML) workloads on AWS. It also gives a high-level view of their legacy and current training and inference architectures.

Last AWS re:Invent, we announced the general availability of Amazon EMR on Amazon Elastic Kubernetes Service (Amazon EKS), a new deployment option for Amazon EMR that allows customers to automate the provisioning and management of Apache Spark on Amazon EKS. With Amazon EMR on EKS, customers can deploy EMR applications on the same Amazon EKS cluster as other types of applications, which allows them to share resources and standardize on a single solution for operating and managing all their applications. Customers running Apache Spark on Kubernetes can migrate to EMR on EKS and take advantage of the performance-optimized runtime, integration with Amazon EMR Studio for interactive jobs, integration with Apache Airflow and AWS Step Functions for running pipelines, and Spark UI for debugging. When customers submit jobs, EMR automatically packages the application into a container with the big data framework and provides prebuilt connectors for integrating with other AWS services. EMR then deploys the application on the EKS cluster and manages running the jobs, logging, and monitoring.

Amazon wants to make it easier to get AI-powered apps up and running on Amazon Web Services. Toward that end, it today launched AWS Deep Learning Containers, a library of Docker images preinstalled with popular deep learning frameworks. "We've done all the hard work of building, compiling, and generating, configuring, optimizing all of these frameworks, so you don't have to," Dr. Matt Wood, general manager of deep learning and AI at AWS, said onstage at the AWS Summit in Santa Clara this morning. "And that means that you do less of the undifferentiated heavy lifting of installing these very, very complicated frameworks and then maintaining them." The new AWS container images in question -- which are preconfigured and validated by Amazon -- support Google's TensorFlow machine learning framework and Apache MXNet, with Facebook's PyTorch and other deep learning frameworks to come.